Anuj Shankar Anuj Shankar - 1 year ago 129
Python Question

Not Getting the Expected Output in AzureML

Background: I am working on a project that aims to classify product reviews into positive and negative using Sentiment Analysis in Azure ML. I got stuck when I was classifying reviews into different departments.

I am basically reading words from csv files and checking whether the review(v: list of sentences) contains these words. If some of these words are found in the review then I am noting the sentence number and pushing it into respective lists( FinanceList, QualityList, LogisticsList ). In the end I am converting the lists to strings and pushing them into a dataframe.

The output is not getting logged for the print statements that I have written in the script in Azure ML.

The values in the dataframe are always turning out to be 0 but when I run the code locally I get the expected output.

Description of First Image: The columns of the dataframe showing 0 values.

Description of Second Image: I have highlighted the expected output that I got locally for the same review which was used in AzureML.

Image 1

Image 2

The things that I have already checked:

  1. The csv files are read properly.

  2. The review contains the words that I am searching.

I am unable to understand where I am going wrong.


import csv
import math
import pandas as pd
import numpy as np

def azureml_main( data, ud):

FinanceDept = []
LogisticsDept = []
QualityDept = []
#Reading from the csv files
with open('.\Script Bundle\\quality1.csv', 'rb') as fin:
reader = csv.reader(fin)
QualityDept = list(reader)

with open('.\Script Bundle\\finance1.csv', 'rb') as f:
reader = csv.reader(f)
FinanceDept = list(reader)

with open('.\Script Bundle\\logistics1.csv', 'rb') as f:
reader = csv.reader(f)
LogisticDept = list(reader)

FinanceList = []
LogisticsList = []
QualityList = []

#Initializing the Lists

rev = data['Data']
v = rev[0].split('.')

print FinanceDept

S = 0
for sentence in v:
S = S + 1
z = sentence.split(' ')
for c in z:
c = c.lower()
if c in FinanceDept and S not in FinanceList:
if c in LogisticsDept and S not in LogisticsList:
if c in QualityDept and S not in QualityList:
#Compute User Reputation Score
Upvotes = int(ud['upvotes'].tolist()[0])
Downvotes = int(ud['downvotes'].tolist()[0])
TotalVotes = max(1,Upvotes+Downvotes)

q = data['Score']

print FinanceList

repScore = float(Upvotes)/TotalVotes
repScore = repScore*float( q[0] )
str1 = ','.join(str(e) for e in FinanceList)
str2 = ','.join(str(e) for e in QualityList)
str3 = ','.join(str(e) for e in LogisticsList)

x = ud['id']

#df = pd.DataFrame( [str(repScore), str1 , str2 , str3 ], columns=[Write the columns])
d = {'id': x[0], 'Score': float(repScore),'Logistics':str3,'Finance':str1,'Quality':str2}
df = pd.DataFrame(data=d, index=np.arange(1))
return df,`

Answer Source

@Anuj Shankar, After my colleague tested, we can read data from CSV files and get the expected results. Please refer to this experience:

1) Input data - It has file which has two csv files similar to you and each csv file includes bag of words related to company.

enter image description here 2) Python script:

# The script MUST contain a function named azureml_main
# which is the entry point for this module.
# The entry point function can contain up to two input arguments:
#   Param<dataframe1>: a pandas.DataFrame
#   Param<dataframe2>: a pandas.DataFrame
import csv
import numpy as np
import pandas as pd

def azureml_main(dataframe1 = None, dataframe2 = None):
    # Execution logic goes here
    #print('Input pandas.DataFrame #1:\r\n\r\n{0}'.format(dataframe1))

    # If a zip file is connected to the third input port is connected,
    # it is unzipped under ".\Script Bundle". This directory is added
    # to sys.path. Therefore, if your zip file contains a Python file
    # you can import it using:
    # import mymodule

    apple = {}
    microsoft = {}
  #Reading from the csv files
    with open('.\Script Bundle\\apple.csv', 'rb') as f:
      reader = csv.reader(f)
      apple = list_to_dict(list(reader)[0])

    with open('.\Script Bundle\\microsoft.csv', 'rb') as f:
      reader = csv.reader(f)
      microsoft = list_to_dict(list(reader)[0])

#    print('hello world' + ' '.join(apple[0]))
    applecount = 0
    microsoftcount = 0

    input = "i want to buy surface which runs on windows"
    splitted_input = input.split(' ')

    for word in splitted_input:
        if word in apple:
            applecount = applecount + 1
        if word in microsoft:
            microsoftcount = microsoftcount + 1

    print("apple bag of words count - " + str(applecount))
    print("microsoft bag of words count - " + str(microsoftcount))
    mydata = [{'input words': len(splitted_input)}, {'applecount':applecount},
    # Return value must be of a sequence of pandas.DataFrame
    return pd.DataFrame(mydata),

def list_to_dict(li):      
    dct = {}  
    for item in li:
        if dct.has_key(item):              
            dct[item] = dct[item] + 1  
            dct[item] = 1  
    return dct  

3) Output - if I consider a string "i want to buy surface which runs on windows". It has 2 words related to microsoft and 0 related to apple which are visualized in below snapshot. enter image description here

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download