Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A couple integrity questions #6834

Open
1 of 3 tasks
andrewgdunn opened this issue Feb 1, 2024 · 11 comments
Open
1 of 3 tasks

A couple integrity questions #6834

andrewgdunn opened this issue Feb 1, 2024 · 11 comments

Comments

@andrewgdunn
Copy link

andrewgdunn commented Feb 1, 2024 •
edited

The bug

feel free to convert this to a discussion, if actual bugs emerge from the discussion it would be better to track them individually

In discussion with @jrasm91 wanted to enumerate a couple integrity related log errors on my instance, these are likely instance specific "holes" that can be fallen into (especially if you were an early adopter). Tooling for examining instance integrity would be fantastic but realizing that it's potentially a lower priority than, say, search at this time.

On this instance all jobs have been re-run fully on the entire dataset.

I have never manipulated the underlying files or database, these errors in state were achieved through likely having a long running instance and not getting on top of these errors when they initially emerged.

Most likely these are all missing files that have "ghosted" over time. Some excerpt errors, will happily dig into each of these in more detail. There appears to be many (~300 within a ~200k) instances of these two types of errors which looks like missing assets or potentially originating file formats that immich isn't prepared for (e.g. VHS conversions of home movies where I may have ffmpeg'ed poorly):

Thumbnail Generation

0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "09b82c91-36c5-461f-8305-e98e68c9a341"
0695508d4f2c }
0695508d4f2c 
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Error: Input file contains unsupported image format
0695508d4f2c     at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
0695508d4f2c     at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:41:14)
0695508d4f2c     at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:121:44)
0695508d4f2c     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
0695508d4f2c     at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:108:28)
0695508d4f2c     at async /usr/src/app/dist/domain/job/job.service.js:116:37
0695508d4f2c     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
0695508d4f2c     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)


0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "e7cbe6b7-8647-4401-84f0-2a299a29a24b"
0695508d4f2c }
0695508d4f2c 
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [MediaRepository] ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
0695508d4f2c   built with gcc 12 (Debian 12.2.0-14)
0695508d4f2c   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
0695508d4f2c   libavutil      58.  2.100 / 58.  2.100
0695508d4f2c   libavcodec     60.  3.100 / 60.  3.100
0695508d4f2c   libavformat    60.  3.100 / 60.  3.100
0695508d4f2c   libavdevice    60.  1.100 / 60.  1.100
0695508d4f2c   libavfilter     9.  3.100 /  9.  3.100
0695508d4f2c   libswscale      7.  1.100 /  7.  1.100
0695508d4f2c   libswresample   4. 10.100 /  4. 10.100
0695508d4f2c   libpostproc    57.  1.100 / 57.  1.100
0695508d4f2c [matroska,webm @ 0x29018120180] max_analyze_duration 5000000 reached at 5024000 microseconds st:1
0695508d4f2c [matroska,webm @ 0x29018120180] Could not find codec parameters for stream 0 (Video: mpeg4, 1 reference frame, none(left), 1920x1080): unspecified pixel format
0695508d4f2c Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
0695508d4f2c upload/library/andrew/2017/2017-01-02/00011.mkv: could not seek to position 0.000
0695508d4f2c Input #0, matroska,webm, from 'upload/library/andrew/2017/2017-01-02/00011.mkv':
0695508d4f2c   Duration: 00:00:00.03, start: 0.000000, bitrate: 226225 kb/s
0695508d4f2c   Stream #0:0(eng): Video: mpeg4, 1 reference frame, none(left), 1920x1080, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn (default)
0695508d4f2c   Stream #0:1: Audio: ac3, 48000 Hz, stereo, fltp, 256 kb/s (default)
0695508d4f2c Stream mapping:
0695508d4f2c   Stream #0:0 -> #0:0 (mpeg4 (native) -> mjpeg (native))
0695508d4f2c Press [q] to stop, [?] for help
0695508d4f2c EOF in input file 0
0695508d4f2c Terminating demuxer thread 0
0695508d4f2c Cannot determine format of input stream 0:0 after EOF
0695508d4f2c Error marking filters as finished
0695508d4f2c [AVIOContext @ 0x29018240180] Statistics: 1168387 bytes read, 5 seeks
0695508d4f2c Conversion failed!
0695508d4f2c

Video Transcoding

Likely ~400 of these on my instance. We've always had pixel devices (or "real" cameras) so I think this may be a different bug than the recent #6337.

0695508d4f2c [Nest] 21  - 02/01/2024, 8:27:10 AM   ERROR [JobService] Error: ffprobe exited with code 1
0695508d4f2c ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
0695508d4f2c   built with gcc 12 (Debian 12.2.0-14)
0695508d4f2c   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
0695508d4f2c   libavutil      58.  2.100 / 58.  2.100
0695508d4f2c   libavcodec     60.  3.100 / 60.  3.100
0695508d4f2c   libavformat    60.  3.100 / 60.  3.100
0695508d4f2c   libavdevice    60.  1.100 / 60.  1.100
0695508d4f2c   libavfilter     9.  3.100 /  9.  3.100
0695508d4f2c   libswscale      7.  1.100 /  7.  1.100
0695508d4f2c   libswresample   4. 10.100 /  4. 10.100
0695508d4f2c   libpostproc    57.  1.100 / 57.  1.100
0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] moov atom not found
0695508d4f2c upload/encoded-video/2350292b-85e4-4e11-9576-1eebd343ccd0/a1/23/a12304f3-e4c2-42c0-8f56-4c1ccd23f0e2-MP.mp4: Invalid data found when processing input
0695508d4f2c 
0695508d4f2c     at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
0695508d4f2c     at ChildProcess.emit (node:events:518:28)
0695508d4f2c     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
0695508d4f2c [Nest] 21  - 02/01/2024, 8:27:10 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "ca4a1c29-9844-4ad1-aabf-e873aee207c5"
0695508d4f2c }

Other Errors

3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:22 PM   ERROR [Error: write EPIPE
3c05918f63e6     at afterWriteDispatched (node:internal/stream_base_commons:160:15)
3c05918f63e6     at writeGeneric (node:internal/stream_base_commons:151:3)
3c05918f63e6     at Socket._writeGeneric (node:net:952:11)
3c05918f63e6     at Socket._write (node:net:964:8)
3c05918f63e6     at doWrite (node:internal/streams/writable:590:12)
3c05918f63e6     at clearBuffer (node:internal/streams/writable:773:7)
3c05918f63e6     at Writable.uncork (node:internal/streams/writable:523:7)
3c05918f63e6     at connectionCorkNT (node:_http_outgoing:955:8)
3c05918f63e6     at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:22 PM   ERROR [Error: write EPIPE
3c05918f63e6     at afterWriteDispatched (node:internal/stream_base_commons:160:15)
3c05918f63e6     at writeGeneric (node:internal/stream_base_commons:151:3)
3c05918f63e6     at Socket._writeGeneric (node:net:952:11)
3c05918f63e6     at Socket._write (node:net:964:8)
3c05918f63e6     at doWrite (node:internal/streams/writable:590:12)
3c05918f63e6     at clearBuffer (node:internal/streams/writable:773:7)
3c05918f63e6     at Writable.uncork (node:internal/streams/writable:523:7)
3c05918f63e6     at connectionCorkNT (node:_http_outgoing:955:8)
3c05918f63e6     at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:23 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:23 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:26 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:26 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:32 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:32 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE

While browsing I see this, maybe nginx headers not configured properly?

3c05918f63e6 [Nest] 3  - 02/01/2024, 12:03:48 PM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

The OS that Immich Server is running on

Centos 9 Stream, Podman, systemd

Version of Immich Server

v1.94.1

Version of Immich Mobile App

v1.94.1.build.121

Platform with the issue

  • Server
  • Web
  • Mobile
@cgeheim
Copy link

cgeheim commented Feb 1, 2024

Same problems here

@dmandrone
Copy link

dmandrone commented Feb 23, 2024 •
edited

Also getting the same issue any time a video is played (see Other Errors section)

@Pheggas
Copy link

Pheggas commented Mar 18, 2024

Bump?? These are serious bugs.

@alextran1502
Copy link
Contributor

@Pheggas, I agree, and we have put more effort into hardening the storage template mechanism. I think you and Andrew was one of the first users of Immich so there could have been artifacts from before we put in the hardening process as well. We keep this open to make sure to revisit this mechanism

cc: @zackpollard

@zackpollard
Copy link
Contributor

Yea, I would be interested if this issue has gotten worse after our storage migration hardening efforts. On the integrity tooling side, having a automated scan of integrity is something that's on my to-do list. As we have the hash of all files, doing this should be pretty simple overall. The problem is there are many things on my to-do list 😅

@Dmoney517
Copy link

I'm seeing the write epipe errors as well. I'm new to the app, is this relates to using the storage template? If I wasn't log watching while setting up and running my imports, I never would have known there was an issue. The videos play and assets load, but it is accompanied by the errors.

@Sid-Sun
Copy link

Sid-Sun commented Apr 6, 2024

I am on v1.101.0
my cluster was set up in August 2023
I was putting labels on my users and doing the migration
even after running full extract metadata job, 3 assets on my server refuse to be migrated:

immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:13 PM   ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration

2 of these have the video transcribe error described here, one does not.

I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on
@zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it.

Thanks <3

@zackpollard
Copy link
Contributor

I am on v1.101.0
my cluster was set up in August 2023
I was putting labels on my users and doing the migration
even after running full extract metadata job, 3 assets on my server refuse to be migrated:

immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:13 PM   ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration

2 of these have the video transcribe error described here, one does not.

I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on
@zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it.

Thanks <3

Hey, the overall idea is a tool that will incrementally scan all assets on the server over time, recomputing the hash of the asset on disk and comparing it to the hash in the database to make sure it hasn't changed. However before we do this work we wanted to make changes to the current repair page to improve that so it works in the background too, as for big instances this page consistently times out as it computes everything when you load the page.

@Sid-Sun
Copy link

Sid-Sun commented Apr 6, 2024

I just discovered there is a repair page XD

I see about 413 untracked files on mine at the moment, I have never modified the underlying storage or DB, is this because of the file integrity changing? Is the idea to similarly show the files which have changed integrity over time?

I'll check out how the repair page works & if I have any ideas, thank you! If there is a discussion open you can link me to, I'll catch up on it.

Minor update: One of the assets of mine which couldn't be moved might be a deleted photo that is in the thrash but I haven't visually confirmed it yet.

This was referenced Apr 7, 2024
Closed
Closed
@brandonkal
Copy link

I am seeing this EPIPE issue as well. New user since 1.91. I've had to wait 25 seconds for the large image thumbnail to load.

@MiguelNdeCarvalho
Copy link

Hey,

I have created a brand new instance and I have imported all my data by using the immich-go CLI utility and when I run Transcode Videos only for Missing items, almost 1200 items, they never get transcoded and always had them back to the queue whenever I click again, here are the logs that I get.

[Nest] 224  - 04/29/2024, 4:05:21 PM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] moov atom not found
/photos/upload/<redacted>/<redacted>.mov: Invalid data found when processing input

    at ChildProcess.<anonymous> (/app/immich/server/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 224  - 04/29/2024, 4:05:21 PM   ERROR [JobService] Object:
{
  "id": "<redacted>"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@andrewgdunn @zackpollard @brandonkal @Pheggas @dmandrone @Sid-Sun @alextran1502 @MiguelNdeCarvalho @Dmoney517 @cgeheim