Continue.dev cannot index two highly similar directories for @Codebase

[JubilantJerry]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Ubuntu 24.04
- Continue version: 0.9.269
- IDE version: VSCode 1.97.2
- Model: Qwen 2.5 7b Instruct
- config:
  
{
  "models": [
    {
      "title": "Qwen 2.5 72b Instruct",
      "model": "qwen2.5:72b-instruct-q8_0",
      "provider": "ollama",
      "apiBase": "https://api.ollama.companyname.com",
      "contextLength": 16384
    },
    {
      "title": "Qwen 2.5 7b Instruct",
      "model": "qwen2.5:7b-instruct-q8_0",
      "provider": "ollama",
      "apiBase": "https://api.ollama.companyname.com",
      "contextLength": 16384
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder 1.5B",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b-base",
    "apiBase": "https://api.ollama.companyname.com"
  },
  "embeddingsProvider": {
    "title": "Nomic Embed Text",
    "provider": "ollama",
    "model": "nomic-embed-text",
    "apiBase": "https://api.ollama.companyname.com"
  },
  "reranker": {
    "name": "huggingface-tei",
    "params": {
      "apiBase": "http://192.168.187.57:11435",
      "truncate": true,
      "truncation_direction": "Right"
    }
  },
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Description

If I open a directory containing identical files as a previously indexed directory, then I will not be able to retrieve any context items with @Codebase in the new directory. It will only include the files that are currently open in the IDE. If the two directories are similar but not fully identical, then all the identical files will be invisible to @Codebase, and only modified files can appear.

To reproduce

Here I provide very precise repro steps, it doesn't have to be done this way.

First, clone this repo: https://github.com/dateutil/dateutil

Then, make a copy of the directory called dateutil_v2.

Open dateutil in VSCode, and follow the steps in this video. At some point in the middle, I open the dateutil_v2 directory.

continue_dev_similar_repo.mp4

I went a step further and looked inside the index.sqlite file, while also debugging to see what query was being run. There, I find that the query includes the filter clause WHERE ... chunk_tags.tag IN ([directory name]). I see no new chunks were being added for the new directory, so there is nothing corresponding to the new directory in the chunks_tags table, but the SQL query is still looking for the new directory name in that table, so nothing is retrieved since everything is filtered out of the WHERE clause.

This sounds to me like a caching optimization is being done to efficiently update the index when a given code base is being updated, but that this caching is also affecting the indexing of new directories. The new directory contents are not indexed since the chunks match cached items, but the search cannot actually use these cached chunks due to the way the SQL queries are made, so nothing can be found in the index.

Log output

log.ts:446  WARN Via 'product.json#extensionEnabledApiProposals' extension 'ms-vscode.vscode-selfhost-test-provider' wants API proposal 'attributableCoverage' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
warn @ log.ts:446
log.ts:446  WARN Via 'product.json#extensionEnabledApiProposals' extension 'ms-python.python' wants API proposal 'contribIssueReporter' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
warn @ log.ts:446
log.ts:446  WARN Via 'product.json#extensionEnabledApiProposals' extension 'ms-python.debugpy' wants API proposal 'contribIssueReporter' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
warn @ log.ts:446
log.ts:446  WARN Via 'product.json#extensionEnabledApiProposals' extension 'redhat.java' wants API proposal 'documentPaste' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
warn @ log.ts:446
log.ts:436  INFO Started local extension host with pid 2914938.
log.ts:436  INFO Settings Sync: Account status changed from uninitialized to unavailable
log.ts:446  WARN Authentication provider continue was not declared in the Extension Manifest.
warn @ log.ts:446
log.ts:436  INFO [perf] Render performance baseline is 17ms
webviewElement.ts:486 An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing.
mountTo @ webviewElement.ts:486
index.js:541 PROFILES:  Array(1)
index.js:541 selecting 3 null
index.js:541 Running Thunk:  local null Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:541 AVAILABLE:  Array(1) local
index.js:541 selecting 3 local
index.js:541 Running Thunk:  local local Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:585 selecting 1 local
index.js:541 Running Thunk:  local local Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:541 PROFILES:  Array(1)
index.js:541 selecting 3 null
index.js:541 Running Thunk:  local null Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:585 CONFIG UPDATE:  Object
index.js:585 selecting 1 local
index.js:541 Running Thunk:  local local Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:585 CONFIG UPDATE:  Object
index.js:585 selecting 1 local
index.js:541 Running Thunk:  local local Array(1)
index.js:541 update selected profile? local local Array(1)
index.js:1102 Skipping file provider due to disabled indexing
index.js:1102 Skipping prompt-files provider due to disabled indexing
index.js:1102 Skipping docs provider due to disabled indexing
console.ts:137 [Extension Host] Table not found in LanceDB filehomejubilantjerryDocumentsCrystalMuseDownloadsReposdateutil_v2mastervectordbOllama2nomic-embed-text500
Ats @ console.ts:137
notificationsAlerts.ts:42 Error getting context items from codebase: Error: {"error":"`texts` cannot be empty","error_type":"Empty"}
c @ notificationsAlerts.ts:42
console.ts:137 [Extension Host] Table not found in LanceDB filehomejubilantjerryDocumentsCrystalMuseDownloadsReposdateutil_v2mastervectordbOllama2nomic-embed-text500
Ats @ console.ts:137
notificationsAlerts.ts:42 Error getting context items from codebase: Error: {"error":"`texts` cannot be empty","error_type":"Empty"}
c @ notificationsAlerts.ts:42
console.ts:137 [Extension Host] Table not found in LanceDB filehomejubilantjerryDocumentsCrystalMuseDownloadsReposdateutil_v2mastervectordbOllama2nomic-embed-text500
Ats @ console.ts:137