Listing Blobs in GCS using java with pagination keep getting the same page of blobs

Sam Tsai :

I am trying to list a large amount of gcs blobs using Java api. Since the number of blobs is large, I tried to use pagination. I am getting the same page repeatedly. The code looked like this

Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();

Page<Blob> allBlobs = storage.list(myBucketName,Storage.BlobListOption.pageSize(5000), Storage.BlobListOption.prefix("some prefix");

while (allBlobs.hasNextPage()) {
   Page<Blob> page = allBlobs.getNextPage();
   for (Blob blob : page.getValues()) {
     .... do something....
   }

}

Looked like I am getting the same page over and over. I looked at the token by allBlobs.getNextPageToken() and the token looked the same all the time. Did I missed something to move the page forward to next one? Isn't getNextPage doing it? The Page interface got only these few methods defined. Did I overlooked?

jterrace :

Use the iterateAll method instead. See example here (copied here for completion):

Page<Blob> blobs =
    storage.list(
        bucketName, BlobListOption.currentDirectory(),
        BlobListOption.prefix(directory));

for (Blob blob : blobs.iterateAll()) {
  // do something with the blob
}