Data_Kid Data_Kid - 1 month ago 9
Python Question

Selecting a rows from a panda frame based on a time that has been converted into datetime format and stripped from a POSIX time stamp, python

I am using the code below to pull data from Google Finance. The timestamp is in POSIX form, so it is converted into data time. When I try to fiter it based on a time criteria (14:35:00), it returns an empty table. I suspect it has to do with the POSIX/ datetime conversion, but have no idea how to resolve it.

def get_intraday_data(symbol, interval_seconds=301, num_days=10):
# Specify URL string based on function inputs.
url_string = 'http://www.google.com/finance/getprices?q={0}'.format(symbol.upper())
url_string += "&i={0}&p={1}d&f=d,o,h,l,c,v".format(interval_seconds,num_days)

# Request the text, and split by each line
r = requests.get(url_string).text.split()

# Split each line by a comma, starting at the 8th line
r = [line.split(',') for line in r[7:]]

# Save data in Pandas DataFrame
df = pd.DataFrame(r, columns=['Datetime','Close','High','Low','Open','Volume'])

# Convert UNIX to Datetime format
df['Datetime'] = df['Datetime'].apply(lambda x: datetime.datetime.fromtimestamp(int(x[1:])))

#Seperate Date and Time
df['Time'],df['Date']= df['Datetime'].apply(lambda x:x.time()), df['Datetime'].apply(lambda x:x.date())

#Convert 'Close','High','Low','Open', deleting 'Volume'

''''df['Close'] = df['Close'].astype('float64')
df['High'] = df['High'].astype('float64')
df['Low'] = df['Low'].astype('float64')
df['Open'] = df['Open'].astype('float64')'''
del df['Volume']
del df['Datetime']

df[['Close','High','Low','Open']] = df[['Close','High','Low','Open']].astype('float64')

# Calculating %Change and Range

df['%pct'] = (df['Close'] - df['Open'])/df['Open']
df['Range'] = df['High'] - df['Low']

#Sort Columns


return df


I have stored the results of this function as NAS

NAS = get_intraday_data('IXIC', interval_seconds=301, num_days= 100)


The Filter Criteria is:

NAS[NAS['Time'] == '14:35:00']


I will appreciate assistance on this.

Answer

I see that you are converting timestamp to datetime incorrectly. You are calling datetime twice.

Replace

df['Datetime'] = df['Datetime'].apply(lambda x: datetime.datetime.fromtimestamp(int(x[1:])))

with

df['Datetime'] = df['Datetime'].apply(lambda x: datetime.fromtimestamp(int(x[1:])))

In 2nd part of your question:

NAS = get_intraday_data('IXIC', interval_seconds=301, num_days= 100)

NAS[NAS['Time'] == '14:35:00']

You care comparing instance of datetime.time with string, which is not correct. Try

NAS[NAS['Time'] == datetime.strptime('14:35:00', '%H:%M:%S').time()]

it should work as expected.

Update:

Running script with suggested changes will display data as:

Close High Low Open Time Date %pct
60 5162.448 5165.124 5162.448 5165.057 14:35:00 2016-07-29 -0.000505
138 5181.768 5183.184 5181.193 5181.404 14:35:00 2016-08-01 0.000070
216 5130.514 5131.933 5130.434 5131.893 14:35:00 2016-08-02 -0.000269
294 5146.608 5146.608 5143.827 5144.788 14:35:00 2016-08-03 0.000354
372 5163.854 5164.154 5162.997 5164.021 14:35:00 2016-08-04 -0.000032
450 5221.624 5221.911 5220.658 5220.789 14:35:00 2016-08-05 0.000160
528 5204.111 5204.240 5202.476 5202.865 14:35:00 2016-08-08 0.000239
. . . 3648 5282.999 5283.017 5279.008 5279.340 14:35:00 2016-10-04 0.000693
3726 5324.450 5325.375 5323.628 5324.129 14:35:00 2016-10-05 0.000060
3804 5310.945 5311.454 5310.194 5310.558 14:35:00 2016-10-06 0.000073
3882 5295.064 5295.080 5292.184 5292.327 14:35:00 2016-10-07 0.000517

Comments