___

Copyright by Pierian Data Inc. For more information, visit us at www.pieriandata.com

Combining DataFrames¶

Full Official Guide (Lots of examples!)¶

https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html ¶

In [213]:

import numpy as np
import pandas as pd

Concatenation¶

Directly "glue" together dataframes.

In [214]:

data_one = {'A': ['A0', 'A1', 'A2', 'A3'],'B': ['B0', 'B1', 'B2', 'B3']}

In [215]:

data_two = {'C': ['C0', 'C1', 'C2', 'C3'], 'D': ['D0', 'D1', 'D2', 'D3']}

In [216]:

one = pd.DataFrame(data_one)

In [217]:

two = pd.DataFrame(data_two)

In [218]:

one

Out[218]:

	A	B
0	A0	B0
1	A1	B1
2	A2	B2
3	A3	B3

In [219]:

two

Out[219]:

	C	D
0	C0	D0
1	C1	D1
2	C2	D2
3	C3	D3

Axis = 0¶

Concatenate along rows¶

In [220]:

axis0 = pd.concat([one,two],axis=0)

In [221]:

axis0

Out[221]:

	A	B	C	D
0	A0	B0	NaN	NaN
1	A1	B1	NaN	NaN
2	A2	B2	NaN	NaN
3	A3	B3	NaN	NaN
0	NaN	NaN	C0	D0
1	NaN	NaN	C1	D1
2	NaN	NaN	C2	D2
3	NaN	NaN	C3	D3

Axis = 1¶

Concatenate along columns¶

In [222]:

axis1 = pd.concat([one,two],axis=1)

In [223]:

axis1

Out[223]:

	A	B	C	D
0	A0	B0	C0	D0
1	A1	B1	C1	D1
2	A2	B2	C2	D2
3	A3	B3	C3	D3

Axis 0 , but columns match up¶

In case you wanted this:

In [224]:

two.columns = one.columns

In [225]:

pd.concat([one,two])

Out[225]:

	A	B
0	A0	B0
1	A1	B1
2	A2	B2
3	A3	B3
0	C0	D0
1	C1	D1
2	C2	D2
3	C3	D3

Merge¶

Data Tables¶

In [226]:

registrations = pd.DataFrame({'reg_id':[1,2,3,4],'name':['Andrew','Bobo','Claire','David']})
logins = pd.DataFrame({'log_id':[1,2,3,4],'name':['Xavier','Andrew','Yolanda','Bobo']})

In [227]:

registrations

Out[227]:

	reg_id	name
0	1	Andrew
1	2	Bobo
2	3	Claire
3	4	David

In [228]:

logins

Out[228]:

	log_id	name
0	1	Xavier
1	2	Andrew
2	3	Yolanda
3	4	Bobo

pd.merge()¶

Merge pandas DataFrames based on key columns, similar to a SQL join. Results based on the how parameter.

In [229]:

help(pd.merge)

Help on function merge in module pandas.core.reshape.merge:

merge(left, right, how: str = 'inner', on=None, left_on=None, right_on=None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes=('_x', '_y'), copy: bool = True, indicator: bool = False, validate=None) -> 'DataFrame'
    Merge DataFrame or named Series objects with a database-style join.
    
    The join is done on columns or indexes. If joining columns on
    columns, the DataFrame indexes *will be ignored*. Otherwise if joining indexes
    on indexes or indexes on a column or columns, the index will be passed on.
    
    Parameters
    ----------
    left : DataFrame
    right : DataFrame or named Series
        Object to merge with.
    how : {'left', 'right', 'outer', 'inner'}, default 'inner'
        Type of merge to be performed.
    
        * left: use only keys from left frame, similar to a SQL left outer join;
          preserve key order.
        * right: use only keys from right frame, similar to a SQL right outer join;
          preserve key order.
        * outer: use union of keys from both frames, similar to a SQL full outer
          join; sort keys lexicographically.
        * inner: use intersection of keys from both frames, similar to a SQL inner
          join; preserve the order of the left keys.
    on : label or list
        Column or index level names to join on. These must be found in both
        DataFrames. If `on` is None and not merging on indexes then this defaults
        to the intersection of the columns in both DataFrames.
    left_on : label or list, or array-like
        Column or index level names to join on in the left DataFrame. Can also
        be an array or list of arrays of the length of the left DataFrame.
        These arrays are treated as if they are columns.
    right_on : label or list, or array-like
        Column or index level names to join on in the right DataFrame. Can also
        be an array or list of arrays of the length of the right DataFrame.
        These arrays are treated as if they are columns.
    left_index : bool, default False
        Use the index from the left DataFrame as the join key(s). If it is a
        MultiIndex, the number of keys in the other DataFrame (either the index
        or a number of columns) must match the number of levels.
    right_index : bool, default False
        Use the index from the right DataFrame as the join key. Same caveats as
        left_index.
    sort : bool, default False
        Sort the join keys lexicographically in the result DataFrame. If False,
        the order of the join keys depends on the join type (how keyword).
    suffixes : tuple of (str, str), default ('_x', '_y')
        Suffix to apply to overlapping column names in the left and right
        side, respectively. To raise an exception on overlapping columns use
        (False, False).
    copy : bool, default True
        If False, avoid copy if possible.
    indicator : bool or str, default False
        If True, adds a column to output DataFrame called "_merge" with
        information on the source of each row.
        If string, column with information on source of each row will be added to
        output DataFrame, and column will be named value of string.
        Information column is Categorical-type and takes on a value of "left_only"
        for observations whose merge key only appears in 'left' DataFrame,
        "right_only" for observations whose merge key only appears in 'right'
        DataFrame, and "both" if the observation's merge key is found in both.
    
    validate : str, optional
        If specified, checks if merge is of specified type.
    
        * "one_to_one" or "1:1": check if merge keys are unique in both
          left and right datasets.
        * "one_to_many" or "1:m": check if merge keys are unique in left
          dataset.
        * "many_to_one" or "m:1": check if merge keys are unique in right
          dataset.
        * "many_to_many" or "m:m": allowed, but does not result in checks.
    
        .. versionadded:: 0.21.0
    
    Returns
    -------
    DataFrame
        A DataFrame of the two merged objects.
    
    See Also
    --------
    merge_ordered : Merge with optional filling/interpolation.
    merge_asof : Merge on nearest keys.
    DataFrame.join : Similar method using indices.
    
    Notes
    -----
    Support for specifying index levels as the `on`, `left_on`, and
    `right_on` parameters was added in version 0.23.0
    Support for merging named Series objects was added in version 0.24.0
    
    Examples
    --------
    
    >>> df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],
    ...                     'value': [1, 2, 3, 5]})
    >>> df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],
    ...                     'value': [5, 6, 7, 8]})
    >>> df1
        lkey value
    0   foo      1
    1   bar      2
    2   baz      3
    3   foo      5
    >>> df2
        rkey value
    0   foo      5
    1   bar      6
    2   baz      7
    3   foo      8
    
    Merge df1 and df2 on the lkey and rkey columns. The value columns have
    the default suffixes, _x and _y, appended.
    
    >>> df1.merge(df2, left_on='lkey', right_on='rkey')
      lkey  value_x rkey  value_y
    0  foo        1  foo        5
    1  foo        1  foo        8
    2  foo        5  foo        5
    3  foo        5  foo        8
    4  bar        2  bar        6
    5  baz        3  baz        7
    
    Merge DataFrames df1 and df2 with specified left and right suffixes
    appended to any overlapping columns.
    
    >>> df1.merge(df2, left_on='lkey', right_on='rkey',
    ...           suffixes=('_left', '_right'))
      lkey  value_left rkey  value_right
    0  foo           1  foo            5
    1  foo           1  foo            8
    2  foo           5  foo            5
    3  foo           5  foo            8
    4  bar           2  bar            6
    5  baz           3  baz            7
    
    Merge DataFrames df1 and df2, but raise an exception if the DataFrames have
    any overlapping columns.
    
    >>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False))
    Traceback (most recent call last):
    ...
    ValueError: columns overlap but no suffix specified:
        Index(['value'], dtype='object')

Inner,Left, Right, and Outer Joins¶

Inner Join¶

Match up where the key is present in BOTH tables. There should be no NaNs due to the join, since by definition to be part of the Inner Join they need info in both tables. Only Andrew and Bobo both registered and logged in.

In [230]:

# Notice pd.merge doesn't take in a list like concat
pd.merge(registrations,logins,how='inner',on='name')

Out[230]:

	reg_id	name	log_id
0	1	Andrew	2
1	2	Bobo	4

In [231]:

# Pandas smart enough to figure out key column (on parameter) if only one column name matches up
pd.merge(registrations,logins,how='inner')

Out[231]:

	reg_id	name	log_id
0	1	Andrew	2
1	2	Bobo	4

In [232]:

# Pandas reports an error if "on" key column isn't in both dataframes
# pd.merge(registrations,logins,how='inner',on='reg_id')

Left Join¶

Match up AND include all rows from Left Table. Show everyone who registered on Left Table, if they don't have login info, then fill with NaN.

In [233]:

pd.merge(registrations,logins,how='left')

Out[233]:

	reg_id	name	log_id
0	1	Andrew	2.0
1	2	Bobo	4.0
2	3	Claire	NaN
3	4	David	NaN

Right Join¶

Match up AND include all rows from Right Table. Show everyone who logged in on the Right Table, if they don't have registration info, then fill with NaN.

In [234]:

pd.merge(registrations,logins,how='right')

Out[234]:

	reg_id	name	log_id
0	1.0	Andrew	2
1	2.0	Bobo	4
2	NaN	Xavier	1
3	NaN	Yolanda	3

Outer Join¶

Match up on all info found in either Left or Right Table. Show everyone that's in the Log in table and the registrations table. Fill any missing info with NaN

In [235]:

pd.merge(registrations,logins,how='outer')

Out[235]:

	reg_id	name	log_id
0	1.0	Andrew	2.0
1	2.0	Bobo	4.0
2	3.0	Claire	NaN
3	4.0	David	NaN
4	NaN	Xavier	1.0
5	NaN	Yolanda	3.0

Join on Index or Column¶

Use combinations of left_on,right_on,left_index,right_index to merge a column or index on each other

In [236]:

registrations

Out[236]:

	reg_id	name
0	1	Andrew
1	2	Bobo
2	3	Claire
3	4	David

In [237]:

logins

Out[237]:

	log_id	name
0	1	Xavier
1	2	Andrew
2	3	Yolanda
3	4	Bobo

In [238]:

registrations = registrations.set_index("name")

In [239]:

registrations

Out[239]:

	reg_id
name
Andrew	1
Bobo	2
Claire	3
David	4

In [240]:

pd.merge(registrations,logins,left_index=True,right_on='name')

Out[240]:

	reg_id	log_id	name
1	1	2	Andrew
3	2	4	Bobo

In [242]:

pd.merge(logins,registrations,right_index=True,left_on='name')

Out[242]:

	log_id	name	reg_id
1	2	Andrew	1
3	4	Bobo	2

Dealing with differing key column names in joined tables¶

In [243]:

registrations = registrations.reset_index()

In [244]:

registrations

Out[244]:

	name	reg_id
0	Andrew	1
1	Bobo	2
2	Claire	3
3	David	4

In [245]:

logins

Out[245]:

	log_id	name
0	1	Xavier
1	2	Andrew
2	3	Yolanda
3	4	Bobo

In [246]:

registrations.columns = ['reg_name','reg_id']

In [247]:

registrations

Out[247]:

	reg_name	reg_id
0	Andrew	1
1	Bobo	2
2	Claire	3
3	David	4

In [248]:

# ERROR
# pd.merge(registrations,logins)

In [249]:

pd.merge(registrations,logins,left_on='reg_name',right_on='name')

Out[249]:

	reg_name	reg_id	log_id	name
0	Andrew	1	2	Andrew
1	Bobo	2	4	Bobo

In [250]:

pd.merge(registrations,logins,left_on='reg_name',right_on='name').drop('reg_name',axis=1)

Out[250]:

	reg_id	log_id	name
0	1	2	Andrew
1	2	4	Bobo

Pandas automatically tags duplicate columns¶

In [255]:

registrations.columns = ['name','id']

In [256]:

logins.columns = ['id','name']

In [257]:

registrations

Out[257]:

	name	id
0	Andrew	1
1	Bobo	2
2	Claire	3
3	David	4

In [258]:

logins

Out[258]:

	id	name
0	1	Xavier
1	2	Andrew
2	3	Yolanda
3	4	Bobo

In [259]:

# _x is for left
# _y is for right
pd.merge(registrations,logins,on='name')

Out[259]:

	name	id_x	id_y
0	Andrew	1	2
1	Bobo	2	4

In [260]:

pd.merge(registrations,logins,on='name',suffixes=('_reg','_log'))

Out[260]:

	name	id_reg	id_log
0	Andrew	1	2
1	Bobo	2	4

	A	B	C	D
0	A0	B0	NaN	NaN
1	A1	B1	NaN	NaN
2	A2	B2	NaN	NaN
3	A3	B3	NaN	NaN
0	NaN	NaN	C0	D0
1	NaN	NaN	C1	D1
2	NaN	NaN	C2	D2
3	NaN	NaN	C3	D3

	A	B	C	D
0	A0	B0	NaN	NaN
1	A1	B1	NaN	NaN
2	A2	B2	NaN	NaN
3	A3	B3	NaN	NaN
0	NaN	NaN	C0	D0
1	NaN	NaN	C1	D1
2	NaN	NaN	C2	D2
3	NaN	NaN	C3	D3

62 KiB Raw Blame History