th4tkh13m
/
udemy-ML
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '8ce038c469'
${ noResults }
udemy-ML/03-Pandas/.ipynb_checkpoints/02-Time-Shifting-checkpoint...

693 lines
18 KiB
Raw Normal View History Unescape

Initial commit
2 years ago
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"___\n",
"\n",
"<a href='http://www.pieriandata.com'><img src='../Pierian_Data_Logo.png'/></a>\n",
"___\n",
"<center><em>Copyright Pierian Data</em></center>\n",
"<center><em>For more information, visit us at <a href='http://www.pieriandata.com'>www.pieriandata.com</a></em></center>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Time Shifting\n",
"\n",
"Sometimes you may need to shift all your data up or down along the time series index. In fact, a lot of pandas built-in methods do this under the hood. This isn't something we'll do often in the course, but it's definitely good to know about this anyways!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"df = pd.read_csv('../Data/starbucks.csv',index_col='Date',parse_dates=True)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-01-02</th>\n",
" <td>38.0061</td>\n",
" <td>6906098</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-05</th>\n",
" <td>37.2781</td>\n",
" <td>11623796</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-06</th>\n",
" <td>36.9748</td>\n",
" <td>7664340</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-07</th>\n",
" <td>37.8848</td>\n",
" <td>9732554</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-08</th>\n",
" <td>38.4961</td>\n",
" <td>13170548</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2015-01-02 38.0061 6906098\n",
"2015-01-05 37.2781 11623796\n",
"2015-01-06 36.9748 7664340\n",
"2015-01-07 37.8848 9732554\n",
"2015-01-08 38.4961 13170548"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2018-12-24</th>\n",
" <td>60.56</td>\n",
" <td>6323252</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-26</th>\n",
" <td>63.08</td>\n",
" <td>16646238</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-27</th>\n",
" <td>63.20</td>\n",
" <td>11308081</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-28</th>\n",
" <td>63.39</td>\n",
" <td>7712127</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-31</th>\n",
" <td>64.40</td>\n",
" <td>7690183</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2018-12-24 60.56 6323252\n",
"2018-12-26 63.08 16646238\n",
"2018-12-27 63.20 11308081\n",
"2018-12-28 63.39 7712127\n",
"2018-12-31 64.40 7690183"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## .shift() forward\n",
"This method shifts the entire date index a given number of rows, without regard for time periods (months & years).<br>It returns a modified copy of the original DataFrame."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-01-02</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-05</th>\n",
" <td>38.0061</td>\n",
" <td>6906098.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-06</th>\n",
" <td>37.2781</td>\n",
" <td>11623796.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-07</th>\n",
" <td>36.9748</td>\n",
" <td>7664340.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-08</th>\n",
" <td>37.8848</td>\n",
" <td>9732554.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2015-01-02 NaN NaN\n",
"2015-01-05 38.0061 6906098.0\n",
"2015-01-06 37.2781 11623796.0\n",
"2015-01-07 36.9748 7664340.0\n",
"2015-01-08 37.8848 9732554.0"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(1).head()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2018-12-24</th>\n",
" <td>61.39</td>\n",
" <td>23524888.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-26</th>\n",
" <td>60.56</td>\n",
" <td>6323252.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-27</th>\n",
" <td>63.08</td>\n",
" <td>16646238.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-28</th>\n",
" <td>63.20</td>\n",
" <td>11308081.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-31</th>\n",
" <td>63.39</td>\n",
" <td>7712127.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2018-12-24 61.39 23524888.0\n",
"2018-12-26 60.56 6323252.0\n",
"2018-12-27 63.08 16646238.0\n",
"2018-12-28 63.20 11308081.0\n",
"2018-12-31 63.39 7712127.0"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# NOTE: You will lose that last piece of data that no longer has an index!\n",
"df.shift(1).tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## .shift() backwards"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-01-02</th>\n",
" <td>37.2781</td>\n",
" <td>11623796.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-05</th>\n",
" <td>36.9748</td>\n",
" <td>7664340.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-06</th>\n",
" <td>37.8848</td>\n",
" <td>9732554.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-07</th>\n",
" <td>38.4961</td>\n",
" <td>13170548.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-08</th>\n",
" <td>37.2361</td>\n",
" <td>27556706.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2015-01-02 37.2781 11623796.0\n",
"2015-01-05 36.9748 7664340.0\n",
"2015-01-06 37.8848 9732554.0\n",
"2015-01-07 38.4961 13170548.0\n",
"2015-01-08 37.2361 27556706.0"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(-1).head()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2018-12-24</th>\n",
" <td>63.08</td>\n",
" <td>16646238.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-26</th>\n",
" <td>63.20</td>\n",
" <td>11308081.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-27</th>\n",
" <td>63.39</td>\n",
" <td>7712127.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-28</th>\n",
" <td>64.40</td>\n",
" <td>7690183.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2018-12-31</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2018-12-24 63.08 16646238.0\n",
"2018-12-26 63.20 11308081.0\n",
"2018-12-27 63.39 7712127.0\n",
"2018-12-28 64.40 7690183.0\n",
"2018-12-31 NaN NaN"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(-1).tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shifting based on Time Series Frequency Code\n",
"\n",
"We can choose to shift <em>index values</em> up or down without realigning the data by passing in a <strong>freq</strong> argument.<br>\n",
"This method shifts dates to the next period based on a frequency code. Common codes are 'M' for month-end and 'A' for year-end. <br>Refer to the <em>Time Series Offset Aliases</em> table from the Time Resampling lecture for a full list of values, or click <a href='http://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases'>here</a>.<br>"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Close</th>\n",
" <th>Volume</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-01-31</th>\n",
" <td>38.0061</td>\n",
" <td>6906098</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-31</th>\n",
" <td>37.2781</td>\n",
" <td>11623796</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-31</th>\n",
" <td>36.9748</td>\n",
" <td>7664340</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-31</th>\n",
" <td>37.8848</td>\n",
" <td>9732554</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-01-31</th>\n",
" <td>38.4961</td>\n",
" <td>13170548</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Close Volume\n",
"Date \n",
"2015-01-31 38.0061 6906098\n",
"2015-01-31 37.2781 11623796\n",
"2015-01-31 36.9748 7664340\n",
"2015-01-31 37.8848 9732554\n",
"2015-01-31 38.4961 13170548"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Shift everything forward one month\n",
"df.shift(periods=1, freq='M').head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more info on time shifting visit http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shift.html<br>\n",
"Up next we'll look at rolling and expanding!"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}