(gtf36) D:\models-master\models\research\object_detection>python model_main.py --pipeline_config_path=mytraining/faster_rcnn_resnet101_coco.config --model_dir=object_detection/mytraining --num_train_steps=20000 --num_eval_steps=1000 --alsologtostderr
2020-03-18 22:51:31.636164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
WARNING:tensorflow:From model_main.py:109: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
W0318 22:51:36.914196 3272 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.
W0318 22:51:36.921151 3272 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0318 22:51:36.922148 3272 model_lib.py:629] Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.
W0318 22:51:36.923146 3272 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.
INFO:tensorflow:Maybe overwriting train_steps: 20000
I0318 22:51:36.924143 3272 config_util.py:488] Maybe overwriting train_steps: 20000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0318 22:51:36.925140 3272 config_util.py:488] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0318 22:51:36.925140 3272 config_util.py:488] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0318 22:51:36.926138 3272 config_util.py:488] Maybe overwriting eval_num_epochs: 1
INFO:tensorflow:Maybe overwriting load_pretrained: True
I0318 22:51:36.926138 3272 config_util.py:488] Maybe overwriting load_pretrained: True
INFO:tensorflow:Ignoring config override key: load_pretrained
I0318 22:51:36.927135 3272 config_util.py:498] Ignoring config override key: load_pretrained
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W0318 22:51:36.928133 3272 model_lib.py:645] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False
I0318 22:51:36.929130 3272 model_lib.py:680] create_estimator_and_inputs: use_tpu False, export_to_tpu False
INFO:tensorflow:Using config: {'_model_dir': 'object_detection/mytraining', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000002327BA0B4E0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0318 22:51:36.930127 3272 estimator.py:212] Using config: {'_model_dir': 'object_detection/mytraining', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000002327BA0B4E0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x000002327BA087B8>) includes params argument, but params are not passed to Estimator.
W0318 22:51:36.932122 3272 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x000002327BA087B8>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0318 22:51:36.933119 3272 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0318 22:51:36.934116 3272 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0318 22:51:36.934116 3272 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0318 22:51:36.952068 3272 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
W0318 22:51:36.967057 3272 module_wrapper.py:139] From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.
W0318 22:51:36.968026 3272 module_wrapper.py:139] From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.
Traceback (most recent call last):
File "D:\models-master\models\research\object_detection\utils\label_map_util.py", line 142, in load_labelmap
text_format.Merge(label_map_string, label_map)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 536, in Merge
descriptor_pool=descriptor_pool)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 590, in MergeLines
return parser.MergeLines(lines, message)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 623, in MergeLines
self._ParseOrMerge(lines, message)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 638, in _ParseOrMerge
self._MergeField(tokenizer, message)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 763, in _MergeField
merger(tokenizer, message, field)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 837, in _MergeMessageField
self._MergeField(tokenizer, sub_message)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 763, in _MergeField
merger(tokenizer, message, field)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 888, in _MergeScalarField
value = tokenizer.ConsumeString()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 1251, in ConsumeString
the_bytes = self.ConsumeByteString()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 1266, in ConsumeByteString
the_list = [self._ConsumeSingleByteString()]
File "E:\Anaconda3\envs\gtf36\lib\site-packages\google\protobuf\text_format.py", line 1285, in _ConsumeSingleByteString
raise self.ParseError('Expected string but found: %r' % (text,))
google.protobuf.text_format.ParseError: 3:7 : Expected string but found: '‘'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "model_main.py", line 109, in <module>
tf.app.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "model_main.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1188, in _train_model_default
input_fn, ModeKeys.TRAIN))
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1025, in _get_features_and_labels_from_input_fn
self._call_input_fn(input_fn, mode))
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1116, in _call_input_fn
return input_fn(**kwargs)
File "D:\models-master\models\research\object_detection\inputs.py", line 504, in _train_input_fn
params=params)
File "D:\models-master\models\research\object_detection\inputs.py", line 607, in train_input
batch_size=params['batch_size'] if params else train_config.batch_size)
File "D:\models-master\models\research\object_detection\builders\dataset_builder.py", line 130, in build
num_additional_channels=input_reader_config.num_additional_channels)
File "D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py", line 319, in __init__
default_value=''),
File "D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py", line 64, in __init__
label_map_proto_file, use_display_name=False)
File "D:\models-master\models\research\object_detection\utils\label_map_util.py", line 172, in get_label_map_dict
label_map = load_labelmap(label_map_path_or_proto)
File "D:\models-master\models\research\object_detection\utils\label_map_util.py", line 144, in load_labelmap
label_map.ParseFromString(label_map_string)
TypeError: a bytes-like object is required, not 'str'这个是因为输入法原因，mylabellmap.pbtxt里面''竟然变成了中文的单引号……= =
修改以后，又出现别的错误
(gtf36) D:\models-master\models\research\object_detection>python model_main.py --pipeline_config_path=mytraining/faster_rcnn_resnet101_coco.config --model_dir=object_detection/mytraining --num_train_steps=20000 --num_eval_steps=1000 --alsologtostderr
2020-03-19 16:11:08.823434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
WARNING:tensorflow:From model_main.py:116: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
E:\Anaconda3\envs\gtf36\lib\site-packages\absl\flags\_validators.py:359: UserWarning: Flag --model_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
'command line!' % flag_name)
E:\Anaconda3\envs\gtf36\lib\site-packages\absl\flags\_validators.py:359: UserWarning: Flag --pipeline_config_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
'command line!' % flag_name)
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
W0319 16:11:11.038187 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.
W0319 16:11:11.046195 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0319 16:11:11.047164 9624 model_lib.py:629] Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.
W0319 16:11:11.048161 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.
INFO:tensorflow:Maybe overwriting train_steps: 20000
I0319 16:11:11.049158 9624 config_util.py:488] Maybe overwriting train_steps: 20000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0319 16:11:11.050156 9624 config_util.py:488] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0319 16:11:11.050156 9624 config_util.py:488] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0319 16:11:11.051153 9624 config_util.py:488] Maybe overwriting eval_num_epochs: 1
INFO:tensorflow:Maybe overwriting load_pretrained: True
I0319 16:11:11.052150 9624 config_util.py:488] Maybe overwriting load_pretrained: True
INFO:tensorflow:Ignoring config override key: load_pretrained
I0319 16:11:11.053147 9624 config_util.py:498] Ignoring config override key: load_pretrained
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W0319 16:11:11.054145 9624 model_lib.py:645] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False
I0319 16:11:11.054145 9624 model_lib.py:680] create_estimator_and_inputs: use_tpu False, export_to_tpu False
INFO:tensorflow:Using config: {'_model_dir': 'object_detection/mytraining', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001B2BA5955F8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0319 16:11:11.055142 9624 estimator.py:212] Using config: {'_model_dir': 'object_detection/mytraining', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001B2BA5955F8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x000001B2BA5A9048>) includes params argument, but params are not passed to Estimator.
W0319 16:11:11.057137 9624 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x000001B2BA5A9048>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0319 16:11:11.060130 9624 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0319 16:11:11.061127 9624 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0319 16:11:11.062124 9624 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0319 16:11:11.083095 9624 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
W0319 16:11:11.096060 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.
W0319 16:11:11.097031 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.
W0319 16:11:11.112017 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0319 16:11:11.115979 9624 dataset_builder.py:72] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
W0319 16:11:11.119968 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0319 16:11:11.121964 9624 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0319 16:11:11.141937 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.
W0319 16:11:12.219030 9624 module_wrapper.py:139] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.
W0319 16:11:18.257913 9624 module_wrapper.py:139] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0319 16:11:18.337697 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
W0319 16:11:20.281496 9624 module_wrapper.py:139] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.
W0319 16:11:21.947013 9624 module_wrapper.py:139] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0319 16:11:22.531479 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.
W0319 16:11:24.394535 9624 module_wrapper.py:139] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
W0319 16:11:24.866273 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
INFO:tensorflow:Calling model_fn.
I0319 16:11:24.878241 9624 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:168: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
W0319 16:11:24.892208 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:168: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:24.900184 9624 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0319 16:11:24.902178 9624 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.
W0319 16:11:27.076363 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:27.083372 9624 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:558: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.
W0319 16:11:27.083372 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:558: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:27.095336 9624 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0319 16:11:27.096340 9624 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
WARNING:tensorflow:From D:\models-master\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.
W0319 16:11:27.504218 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\core\minibatch_sampler.py:85: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
W0319 16:11:27.534168 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\core\minibatch_sampler.py:85: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\spatial_transform_ops.py:419: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
W0319 16:11:27.680746 9624 deprecation.py:506] From D:\models-master\models\research\object_detection\utils\spatial_transform_ops.py:419: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:191: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.
W0319 16:11:27.694739 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:191: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:27.695707 9624 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
W0319 16:11:27.893178 9624 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:27.896171 9624 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0319 16:11:27.909164 9624 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
W0319 16:11:27.929084 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2768: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
W0319 16:11:27.931077 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2768: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.
W0319 16:11:27.934069 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\utils\variables_helper.py:142: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.
W0319 16:11:27.945040 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\utils\variables_helper.py:142: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:353: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.
W0319 16:11:27.950027 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:353: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\core\losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.
W0319 16:11:29.163780 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\core\losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\core\losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.
W0319 16:11:29.165776 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\core\losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\core\losses.py:350: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See `tf.nn.softmax_cross_entropy_with_logits_v2`.
W0319 16:11:29.198688 9624 deprecation.py:323] From D:\models-master\models\research\object_detection\core\losses.py:350: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See `tf.nn.softmax_cross_entropy_with_logits_v2`.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2710: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
W0319 16:11:29.353273 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2710: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:380: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
W0319 16:11:29.355270 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:380: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\builders\optimizer_builder.py:58: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.
W0319 16:11:29.361253 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\builders\optimizer_builder.py:58: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:398: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.
W0319 16:11:29.363248 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:398: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:408: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
W0319 16:11:29.365241 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:408: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:515: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
W0319 16:11:32.505842 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:515: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:519: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.
W0319 16:11:32.977581 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:519: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.
WARNING:tensorflow:From D:\models-master\models\research\object_detection\model_lib.py:520: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.
W0319 16:11:32.978580 9624 module_wrapper.py:139] From D:\models-master\models\research\object_detection\model_lib.py:520: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.
INFO:tensorflow:Done calling model_fn.
I0319 16:11:32.980573 9624 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0319 16:11:32.981571 9624 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0319 16:11:35.675369 9624 monitored_session.py:240] Graph was finalized.
2020-03-19 16:11:35.677074: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-03-19 16:11:35.892180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-03-19 16:11:35.957760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: TITAN Xp COLLECTORS EDITION major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
2020-03-19 16:11:35.961961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-03-19 16:11:35.982751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-03-19 16:11:36.001410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2020-03-19 16:11:36.020025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2020-03-19 16:11:36.044774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2020-03-19 16:11:36.064733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2020-03-19 16:11:36.086417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-03-19 16:11:36.091019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-19 16:11:36.513192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-19 16:11:36.516283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-03-19 16:11:36.517947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-03-19 16:11:36.520930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9586 MB memory) -> physical GPU (device: 0, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from object_detection/mytraining\model.ckpt-0
I0319 16:11:36.530086 9624 saver.py:1284] Restoring parameters from object_detection/mytraining\model.ckpt-0
WARNING:tensorflow:From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
W0319 16:11:38.558658 9624 deprecation.py:323] From E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
INFO:tensorflow:Running local_init_op.
I0319 16:11:39.229864 9624 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0319 16:11:39.517096 9624 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into object_detection/mytraining\model.ckpt.
I0319 16:11:46.600153 9624 basic_session_run_hooks.py:606] Saving checkpoints for 0 into object_detection/mytraining\model.ckpt.
2020-03-19 16:11:54.404039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-03-19 16:11:58.310220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-03-19 16:11:59.664412: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2020-03-19 16:11:59.673749: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(*args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D}}]]
[[Loss/RPNLoss/Loss/huber_loss/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/ExpandDims_1/_7529]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "model_main.py", line 116, in <module>
tf.app.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "model_main.py", line 112, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1195, in _train_model_default
saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\six.py", line 703, in reraise
raise value
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
run_metadata_ptr)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
run_metadata)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D (defined at E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
[[Loss/RPNLoss/Loss/huber_loss/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/ExpandDims_1/_7529]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D (defined at E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D':
File "model_main.py", line 116, in <module>
tf.app.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "model_main.py", line 112, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1191, in _train_model_default
features, labels, ModeKeys.TRAIN, self.config)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1149, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "D:\models-master\models\research\object_detection\model_lib.py", line 308, in model_fn
features[fields.InputDataFields.true_image_shape])
File "D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 836, in predict
prediction_dict = self._predict_first_stage(preprocessed_inputs)
File "D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 890, in _predict_first_stage
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 1330, in _extract_rpn_feature_maps
preprocessed_inputs)
File "D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 1355, in _extract_proposal_features
scope=self.first_stage_feature_extractor_scope))
File "D:\models-master\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 169, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "D:\models-master\models\research\object_detection\models\faster_rcnn_resnet_v1_feature_extractor.py", line 143, in _extract_proposal_features
scope=var_scope)
File "D:\models-master\models\research\slim\nets\resnet_v1.py", line 343, in resnet_v1_101
reuse=reuse, scope=scope)
File "D:\models-master\models\research\slim\nets\resnet_v1.py", line 236, in resnet_v1
net = resnet_utils.conv2d_same(net, 64, 7, stride=2, scope='conv1')
File "D:\models-master\models\research\slim\nets\resnet_utils.py", line 124, in conv2d_same
rate=rate, padding='VALID', scope=scope)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py", line 1159, in convolution2d
conv_dims=2)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py", line 1057, in convolution
outputs = layer.apply(inputs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 1700, in apply
return self.__call__(inputs, *args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\layers\base.py", line 548, in __call__
outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 854, in __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\impl\api.py", line 234, in wrapper
return converted_call(f, options, args, kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\impl\api.py", line 439, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\autograph\impl\api.py", line 330, in _call_unconverted
return f(*args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\keras\layers\convolutional.py", line 197, in call
outputs = self._convolution_op(inputs, self.kernel)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1134, in __call__
return self.conv_op(inp, filter)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 639, in __call__
return self.call(inp, filter)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 238, in __call__
name=self.name)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2010, in conv2d
name=name)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1071, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "E:\Anaconda3\envs\gtf36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()定位到第一次出现Traceback (most recent call last):时的上面一行
2020-03-19 16:11:59.673749: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
原因是cudnn版本太低，从7.4升级到7.6就可以了
p.s.:搜索了一整天tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D}}]]
[[Loss/RPNLoss/Loss/huber_loss/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/ExpandDims_1/_7529]] 我人傻了还没解决，还是得先仔细看自己的报错哪里有问题啊……
