Моя работа по обучению на платформе AI на платформе Google Cloud, похоже, выполняется, но не требует использования процессора. Программа не завершается, но выдает несколько ошибок при первом запуске задания. Они выглядят так, как показано ниже
INFO 2020-06-05 04:33:38 +0000 master-replica-0 Create CheckpointSaverHook.
ERROR 2020-06-05 04:33:38 +0000 master-replica-0 I0605 04:33:38.890919 139686838036224 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO 2020-06-05 04:33:41 +0000 worker-replica-0 Graph was finalized.
ERROR 2020-06-05 04:33:41 +0000 worker-replica-0 I0605 04:33:41.006648 140712303798016 monitored_session.py:240] Graph was finalized.
INFO 2020-06-05 04:33:41 +0000 worker-replica-4 Graph was finalized.
ERROR 2020-06-05 04:33:41 +0000 worker-replica-4 I0605 04:33:41.482944 139947128342272 monitored_session.py:240] Graph was finalized.
INFO 2020-06-05 04:33:41 +0000 worker-replica-2 Graph was finalized.
ERROR 2020-06-05 04:33:41 +0000 worker-replica-2 I0605 04:33:41.927765 140284058486528 monitored_session.py:240] Graph was finalized.
INFO 2020-06-05 04:33:41 +0000 master-replica-0 Graph was finalized.
ERROR 2020-06-05 04:33:41 +0000 master-replica-0 I0605 04:33:41.995326 139686838036224 monitored_session.py:240] Graph was finalized.
INFO 2020-06-05 04:33:42 +0000 master-replica-0 Restoring parameters from gs://lasertagger_v1/output/models/wikisplit_experiment_name_2/model.ckpt-0
ERROR 2020-06-05 04:33:42 +0000 master-replica-0 I0605 04:33:42.216852 139686838036224 saver.py:1284] Restoring parameters from gs://lasertagger_v1/output/models/wikisplit_experiment_name_2/model.ckpt-0
INFO 2020-06-05 04:33:43 +0000 worker-replica-3 Done calling model_fn.
ERROR 2020-06-05 04:33:43 +0000 worker-replica-3 I0605 04:33:43.411592 140653000845056 estimator.py:1150] Done calling model_fn.
INFO 2020-06-05 04:33:43 +0000 worker-replica-3 Create CheckpointSaverHook.
ERROR 2020-06-05 04:33:43 +0000 worker-replica-3 I0605 04:33:43.413079 140653000845056 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO 2020-06-05 04:33:44 +0000 worker-replica-1 Done calling model_fn.
ERROR 2020-06-05 04:33:44 +0000 worker-replica-1 I0605 04:33:44.139685 140410730743552 estimator.py:1150] Done calling model_fn.
INFO 2020-06-05 04:33:44 +0000 worker-replica-1 Create CheckpointSaverHook.
ERROR 2020-06-05 04:33:44 +0000 worker-replica-1 I0605 04:33:44.141169 140410730743552 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO 2020-06-05 04:33:47 +0000 worker-replica-1 Graph was finalized.
ERROR 2020-06-05 04:33:47 +0000 worker-replica-1 I0605 04:33:47.280014 140410730743552 monitored_session.py:240] Graph was finalized.
INFO 2020-06-05 04:33:47 +0000 worker-replica-3 Graph was finalized.
ERROR 2020-06-05 04:33:47 +0000 worker-replica-3 I0605 04:33:47.335122 140653000845056 monitored_session.py:240] Graph was finalized.
За каждым сообщением INFO следует сообщение ERROR, и я не понимаю, что происходит с этим учебным заданием. Спасибо!
Ниже приведены более подробные сообщения об ошибках:
2020-06-05 13:12:50.583 EDT
worker-replica-4
I0605 17:12:50.583258 140104498276096 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
{
insertId: "o5flw8f1urq2q"
jsonPayload: {
created: 1591377170.5835383
levelname: "ERROR"
lineno: 328
message: "I0605 17:12:50.583258 140104498276096 basic_session_run_hooks.py:541] Create CheckpointSaverHook."
pathname: "/runcloudml.py"
}
labels: {
compute.googleapis.com/resource_id: "2069730006064940177"
compute.googleapis.com/resource_name: "gke-cml-0605-170056-7fb-n1-highmem-96-9990517e-rvlx"
compute.googleapis.com/zone: "us-east1-c"
ml.googleapis.com/job_id/log_area: "root"
ml.googleapis.com/trial_id: ""
}
logName: "projects/smart-content-summary/logs/worker-replica-4"
receiveTimestamp: "2020-06-05T17:13:00.962017815Z"
resource: {
labels: {…}
type: "ml_job"
}
severity: "ERROR"
timestamp: "2020-06-05T17:12:50.583538292Z"
}
