I have a table that I need to parse, specifically it is a school schedule with 4 blocks of time, and 5 blocks of days for every week. I've attempted to parse it, but honestly have not gotten very far because I am stuck with how to deal with rowspan and colspan attributes, because they essentially mean there is a lack of data that I need to continue.
As an example of what I want to do, here's a table:
<tr>
<td colspan="2" rowspan="4">#1</td>
<td rowspan="4">#2</td>
<td rowspan="2">#3</td>
<td rowspan="2">#4</td>
</tr>
<tr>
</tr>
<tr>
<td rowspan="2">#5</td>
<td rowspan="2">#6</td>
</tr>
<tr>
</tr>
[[1,1,2,3,4],
[1,1,2,3,4],
[1,1,2,5,6],
[1,1,2,5,6]]
[1,2,3,4,5,6]
<table>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td rowspan="4">Thing</td>
<td> </td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
</table>
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', ' ', ' ']
[' ', ' ', ' ', ' ', ' ']
[' ', ' ', ' ', ' ', ' ']
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']
<table>
<tr>
<td> </td>
<td rowspan="2"> DMAT Aud. 6 </td>
<td rowspan="4"> Exam</td>
<td rowspan="2"> DMAT Aud. 7</td>
<td> </td>
</tr>
<tr>
<td> </td>
<td rowspan="2"> CART Aud. 4</td>
</tr>
<tr>
<td> </td>
<td rowspan="2"> CART Aud. 4</td>
<td rowspan="2"> OOP Aud. 7</td>
</tr>
<tr>
<td> </td>
<td> </td>
</tr>
</table>
[
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' '],
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' ']
]
[
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' '],
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' CART Aud. 4', ' OOP Aud. 7'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' ']
]
UPDATE (removed previous function)
UPDATE2 fixed and simplified.
My first function was wrong. Here's another one, it's working but needs tests:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from collections import defaultdict
def table_to_list(table):
dct = table_to_2d_dict(table)
return list(iter_2d_dict(dct))
def table_to_2d_dict(table):
result = defaultdict(lambda : defaultdict(unicode))
for row_i, row in enumerate(table.xpath('./tr')):
for col_i, col in enumerate(row.xpath('./td|./th')):
colspan = int(col.get('colspan', 1))
rowspan = int(col.get('rowspan', 1))
col_data = col.text_content()
while row_i in result and col_i in result[row_i]:
col_i += 1
for i in range(row_i, row_i + rowspan):
for j in range(col_i, col_i + colspan):
result[i][j] = col_data
return result
def iter_2d_dict(dct):
for i, row in sorted(dct.items()):
cols = []
for j, col in sorted(row.items()):
cols.append(col)
yield cols
if __name__ == '__main__':
import lxml.html
from pprint import pprint
doc = lxml.html.parse('tables.html')
for table_el in doc.xpath('//table'):
table = table_to_list(table_el)
pprint(table)
tables.html:
<table border="1">
<tr>
<td>1 </td>
<td>1 </td>
<td>1 </td>
<td rowspan="4">Thing</td>
<td>1 </td>
</tr>
<tr>
<td>2 </td>
<td>2 </td>
<td>2 </td>
<td>2 </td>
</tr>
<tr>
<td>3 </td>
<td>3 </td>
<td>3 </td>
<td>3 </td>
</tr>
<tr>
<td>4 </td>
<td>4 </td>
<td>4 </td>
<td>4 </td>
</tr>
</table>
<table border="1">
<tr>
<td colspan="2" rowspan="4">#1</td>
<td rowspan="4">#2</td>
<td rowspan="2">#3</td>
<td rowspan="2">#4</td>
</tr>
<tr></tr>
<tr>
<td rowspan="2">#5</td>
<td rowspan="2">#6</td>
</tr>
<tr></tr>
</table>
Output:
[['1 ', '1 ', '1 ', 'Thing', '1 '],
['2 ', '2 ', '2 ', 'Thing', '2 '],
['3 ', '3 ', '3 ', 'Thing', '3 '],
['4 ', '4 ', '4 ', 'Thing', '4 ']]
[['#1', '#1', '#2', '#3', '#4'],
['#1', '#1', '#2', '#3', '#4'],
['#1', '#1', '#2', '#5', '#6'],
['#1', '#1', '#2', '#5', '#6']]