Selenium: How to avoid StaleElementReferenceException when looping through a set of elements?

user3217883 :

I have a page that contains a bunch of tables. I loop through the tables in the outer loop and then loop through each row in the table in the inner loop. It all works fine. But some of the pages have a Next button. When I add code to click that after completing the page, then I start getting StaleElementReferenceException while looping through the rows of a table.

Here is the code:

WebDriverWait wait1 = new WebDriverWait(driver, 10000);
WebElement maxPage = null;
WebElement auctionsWaitingDiv = driver.findElement(By.cssSelector("div[class='Head_W']"));
if (auctionsWaitingDiv.isDisplayed() == false) return properties;

try {
    maxPage = wait1.until(ExpectedConditions.visibilityOfElementLocated(By.id("maxWA")));
} catch (TimeoutException ex) {
    return properties;
}

Integer maxPageNo = 1;
if (!maxPage.getText().isEmpty()) 
    maxPageNo = Integer.parseInt(maxPage.getText());
for (int i = 1; i <= maxPageNo; i++) {
    driver.findElement(By.cssSelector("div[id='Area_W']"));    //only look at Auctions Waiting section
    WebDriverWait wait2 = new WebDriverWait(driver, 10000);
    List<WebElement> tables = null;
    try {
        tables = wait2.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("table[class='ad_tab']")));
    } catch (TimeoutException ex) {
        System.out.println("table not found in allotted time");
        return properties;
    } catch (StaleElementReferenceException ex) {
        System.out.println("returning due to StaleElementReferenceException");
        return properties;

    }
    for (WebElement table: tables) {  
        List<String> propAttributes = new ArrayList<>();

        // StaleElementReferenceException: The element reference of
        // <table class="ad_tab"> is stale; either the element is no
        // longer attached to the DOM, it is not in the current
        // frame context, or the document has been refreshed
        List<WebElement> rows = table.findElements(By.cssSelector("tr"));

        String parcelLink = "";
        for (WebElement row : rows) { 
            WebElement key = row.findElement(By.cssSelector("th"));
            WebElement val = row.findElement(By.cssSelector("td"));
            String keyVal = key.getText() + val.getText();
            propAttributes.add(keyVal);
            if (key.getText().equals("Parcel ID:")) {
                WebElement a = val.findElement(By.cssSelector("a"));
                parcelLink = a.getAttribute("href");
            }
        }
    }
    driver.findElement(By.xpath(".//*[@class='PageRight']")).click();  //click the "Next" button
}

What I don't understand is why the stale element is happening at all? The page is not changing during the loop and I've waited until all elements have been fetched. How to avoid the StaleElementReferenceException?

Edit: The last stack trace shows it is happening in this line:

List<WebElement> rows = table.findElements(By.cssSelector("tr"));

and the error message above it shows:

SEVERE: null

org.openqa.selenium.StaleElementReferenceException: The element reference of <table class="ad_tab"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

user3217883 :

Well after tearing my hair out for a day, I finally realized what was happening. It should have been obvious to me. When the "Next" button is clicked, it takes some time for the new page to load. By simply adding a delay, the new DOM is loaded and processing begins on it, not again on the previous one!

            driver.findElement(By.xpath(".//*[@class='PageRight']")).click();
        try {
            Thread.sleep(4000); //provide some time for the page to load before processing it
        } catch (InterruptedException ex) {
            Logger.getLogger(RealAuction.class.getName()).log(Level.SEVERE, null, ex);
        }

Now it runs to completion with no StaleElementReferenceException.