ウェブスクレイピングは、すべてのテーブルを取得できません

ミシェルMetran：

私はBeautifulSoupとSeleniumを使用してテーブルを取るコードを書きました。

しかし、テーブルの一部のみが得られます。アクセスしたときに表示されない行と列のウェブサイトは、スープのオブジェクトによって得られていません。

私はこの問題は抜粋に発生することを確信しています WebDriverWait(driver, 10).until (EC.visibility_of_element_located((By.ID,"contenttabledivjqxGrid")))

...（私はセレンと日付を変更する前に、このテーブルのすべての行と列をロードすることである）私は他のいくつかの選択肢を試したが、どれも私に期待される結果を与えませんでした。

コードに従ってください：

import os
import time
from selenium import webdriver
from bs4 import BeautifulSoup

​from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options

# Escolhe o driver Firefox com Profile e Options
driver = webdriver.FirefoxProfile()
driver.set_preference('intl.accept_languages', 'pt-BR, pt')
driver.set_preference('browser.download.folderList', '2')
driver.set_preference('browser.download.manager.showWhenStarting', 'false')
driver.set_preference('browser.download.dir', 'dwnd_path')
driver.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/octet-stream,application/vnd.ms-excel')

options = Options()
options.headless = False

driver = webdriver.Firefox(firefox_profile=driver, options=options)

# Cria um driver

site = 'http://mananciais.sabesp.com.br/HistoricoSistemas'
driver.get(site)


WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.ID,"contenttabledivjqxGrid")))
soup = BeautifulSoup(driver.page_source, 'html.parser')

# Cabeçalho
header = soup.find_all('div', {'class': 'jqx-grid-column-header'})
for i in header:
    print(i.get_text())


# Seleciona as relevantes
head = []
for i in header:
    if i.get_text().startswith(('Represa', 'Equivalente')):
        print('Excluído: ' + i.get_text())
    else:
        print(i.get_text())
        head.append(i.get_text())

print('-'*70)
print(head)
print('-'*70)
print('Número de Colunas: ' + str(len(head)))

# Valores
data = soup.find_all('div', {'class': 'jqx-grid-cell'})
values = []
for i in data:
    print(i.get_text())
    values.append(i.get_text())


import numpy as np
import pandas as pd

# Convert data to numpy array
num = np.array(values)

# Currently its shape is single dimensional
n_rows = int(len(num)/len(head))
n_cols = int(len(head))
reshaped = num.reshape(n_rows, n_cols)

# Construct Table
pd.DataFrame(reshaped, columns=head)

私はちょうど水文学だし、この貯水池データを取得します。いくつかのいずれかが私を助けることができますか？

私の結果テーブルには、今のところ、この次のとおりです。

EnriqueBet：

I just checked the website. In Firefox if you go to Developer Tools > Network and if you check the file with name "0" you will notice that the response of that file is a JSON file with all the information that you need (Image 1). In order to get this information you will have to follow the request Headers (Image 2)

Image 1: Request Response

Image 2: Request Headers

You will need to perform a "GET" request to the website with these headers and the response if it is accepted, will be a JSON with all your data. Bear in mind that some request might ask for a cookie header, which you will need to be obtained before performing the request.

I don't know Beatutiful Soup really well, but I know that this is achievable with Scrapy or with the Request Library. I am pretty sure this will point you to the right direction.