Oppia-cron.sh failing

hakimks · June 24, 2022, 6:05pm

Hi @alex
The oppia cron task has failed to run eversince I upgraded the server to v0.14.0. Below is the error output when I try to run it manually.

/home/staging/env/lib/python3.8/site-packages/django/apps/registry.py:91: RemovedInDjango41Warning: 'sass_processor' defines default_app_config = 'sass_processor.apps.SassProcessorConfig'. Django now detects this configuration automatically. You can remove default_app_config.
  app_config = AppConfig.create(entry)
Last tracker processed: 46611
Newest tracker: 46636

4 different user/courses to process.
processing user/course trackers... (1/4)
processing user/course trackers... (2/4)
processing user/course trackers... (3/4)
processing user/course trackers... (4/4)
6 different courses/dates/types to process.
1
2
3
4
5
6
1 different search/dates to process.
2 different user/points to process.
9 different tracker courses/dates/types to process.
1/9) 2022-06-15 00:00:00+03:00 55 - testuser download: 1
2/9) 2022-06-15 00:00:00+03:00 55 - testuser None: 0
Traceback (most recent call last):
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
summary.models.user_course_daily_summary.DoesNotExist: UserCourseDailySummary matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/home/staging/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/home/staging/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/home/staging/env/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1048, "Column 'type' cannot be null")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "django-oppia/manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "/home/staging/env/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/staging/env/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/staging/env/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/staging/env/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/staging/django-oppia/summary/management/commands/update_summaries.py", line 54, in handle
    self.update_summaries(last_tracker_pk, last_points_pk)
  File "/home/staging/django-oppia/summary/management/commands/update_summaries.py", line 86, in update_summaries
    self.update_user_course_daily_stats(last_tracker_pk, newest_tracker_pk)
  File "/home/staging/django-oppia/summary/management/commands/update_summaries.py", line 181, in update_user_course_daily_stats
    self.update_daily_stats('tracker', 'tracked', last_tracker_pk, newest_tracker_pk)
  File "/home/staging/django-oppia/summary/management/commands/update_summaries.py", line 204, in update_daily_stats
    stats, created = UserCourseDailySummary.objects.get_or_create(
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/staging/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 78, in execute
    raise IntegrityError(*tuple(e.args))
django.db.utils.IntegrityError: (1048, "Column 'type' cannot be null")

alex · June 27, 2022, 5:31am

Hi @hakimks

I’m not exactly sure what might be going wrong here, as it’s not something we’ve seen on our dev or live servers.

The not null property on the type column hasn’t been changed since the table/model was created (in v0.13.1); none of the table structure has been modified since it was created in that version.

A couple of things to check…

Which version did you upgrade from? if you skipped a couple of versions, you’ll likely need to run the update_summaries --fromstart (as this was needed for both 0.13.0 and 0.13.1), although the error might still occur.
The type info comes directly from the type column on the tracker table, so would be good to have a look at the tracker table data and see if there are many rows where the type is none. On our live Oppia we have some tracker rows which have type as none, but these are all very old trackers (from over 7 years ago), so I wonder if there is some very old data (or test data, as looks to be a test user) that’s causing the problem.

If it’s just test data in the tracker table, then likely these can be removed, and see if the cron then runs ok. In the meantime I’ll add a Jira issue to make sure the cron doesn’t just fail, it should skip over this tracker record.
Cheers,
Alex

hakimks · June 27, 2022, 10:29am

Hi @alex
I upgraded from v0.13.1, and I had successfully run the update_sumaries–from start command when I had just upgraded to v0.13.1.

When I tried to re-run the update_sumaries–from start command, its giving me an error this time ( self.model.DoesNotExist(
summary.models.user_course_daily_summary.DoesNotExist: UserCourseDailySummary matching query does not exist.)
.
And I have checked the oppia_tracker tabel and there are some few rows with the ‘type’ missing. I will try to remove these rows and try running the cron task.

alex · June 28, 2022, 5:16am

Hi @hakimks

Actually, before you remove those rows, please can you email me a copy of the db - as it will help a lot when we’re fixing this issue. If you’ve already removed them, then hopefully you can get it from a backup?
Thanks,
Alex

hakimks · June 28, 2022, 7:22am

Hi @alex
I have shared the Db file with you via email.

alex · June 28, 2022, 2:37pm

Thanks @hakimks … that made things easier!

I’ve got the fix done now and so will be in this sprint release (v0.14.1)