Exclude a word if it is present in an array of words
filter out common word before adding word into words list.
I made the fiter function as below:
int isCommonWord(char * word)
{
int i = 0;
for (i = 0; i < NUMBER_OF_STRING; i++) {
if (strcmp(commonWords[i], word) == 0) return 1;
}
return 0;
}
And, filter out word before adding to words array. Please refer the 2nd line of the code what i modified as below:
if (isunique) { /* if unique, add to array, increment index */
if (!isCommonWord(word)) {
if (index == max_words) { /* is realloc needed? */
/* always use a temporary pointer with realloc */
void *tmp = realloc(words, 2 * max_words * sizeof *words);
if (!tmp) { /* validate every allocation */
perror("realloc-words");
break; /* don't exit, original data still valid */
}
words = (words_t *)tmp; /* assign reallocated block to words */
/* (optional) set all new memory to zero */
memset(words + max_words, 0, max_words * sizeof *words);
max_words *= 2; /* update max_words to reflect new limit */
}
memcpy(words[index].word, word, len + 1); /* have len */
if (iscap) /* if cap flag set */
words[index].cap = iscap; /* set capital flag in struct */
words[index++].count++; /* increment count & index */
}
}
I think The result is correct as below:
Enter file path: cars.txt
Occurrences of all distinct words with Cap in file:
2 Motor
8 Cars
1 German
1 Karl
2 Benz
1 Patent-motorwagen
1 Model
1 T
1 American
1 Ford
1 Company
1 Western
1 Europe
1 Electric
2 Road
1 People's
1 China
1 India
A slightly more efficient way would be to use a single call to strstr rather than attempting to compare against every one of the top 100 most common words. Since you know the 100 most common words, and they will not change, you can easily determine the longest of the is 7-characters. In other words, you only need to test whether word is one of the most common if it is less than:
#define TOP_LEN 8 /* longest string in TOP100 + nul-character */
Since the words do not change, you can go ahead and:
const char TOP100[] = " the be to of and a in that have i it for not on with"
" he as you do at this but his by from they we say her she or"
" an will my one all would there their what so up out if about"
" who get which go me when make can like time no just him know"
" take people into year your good some could them see other"
" than then now look only come its over think also back after"
" use two how our work first well way even new want because"
" any these give day most us ";
(note: the space before and the space after each word which allows you to create a teststr to search for with strstr by including a space on either side of your word. 'I' has been converted to lowercase to work after your strlwr (word);)
(also note: you could also use a constant literal with #define TOP100 " the ... us ", but it would wrap and scroll horribly off the page here -- up to you)
With your constant string of the 100 most common words, the only addition needed is:
...
strlwr (word); /* convert word to lowercase */
/* check against 100 most common words (TOP100) */
if (len < TOP_LEN) { /* word less than TOP_LEN? */
char teststr[TOP_LEN * 2]; /* buffer for " word " */
sprintf (teststr, " %s ", word); /* create teststr */
if (strstr (TOP100, teststr)) /* check if in TOP100 */
continue; /* if so, get next word */
}
...
You see above, you check if the word is 7-characters or less (otherwise there is no need to check against the most common). You then declare a teststr to hold you string with a space at each end. (since the longest common word in 7-char, then 7-char plus 2-spaces is 9-char, plus the nul-character is 10, so 16-char is more than adequate here.)
A simple call to sprintf is all that is needed to put the spaces at each end of word, and then a single call to strstr is all that is needed to see if word is within the top 100 most common words. If it is, no need to go further, just continue and get the next word.
Putting it altogether in your code you would have:
/**
* C program to count occurrences of all words in a file.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <limits.h>
#define MAX_WORD 20000 /* max word size */
#define MAX_WORDS 8 /* initial number of struct to allocate */
#define TOP_LEN 8 /* longest string in TOP100 */
#ifndef PATH_MAX
#define PATH_MAX 2048 /* max path (defined for Linux in limits.h) */
#endif
const char TOP100[] = " the be to of and a in that have i it for not on with"
" he as you do at this but his by from they we say her she or"
" an will my one all would there their what so up out if about"
" who get which go me when make can like time no just him know"
" take people into year your good some could them see other"
" than then now look only come its over think also back after"
" use two how our work first well way even new want because"
" any these give day most us ";
typedef struct { /* use a struct to hold */
char word[MAX_WORD]; /* lowercase word, and */
int cap, count; /* if it appeast capitalized, and its count */
} words_t;
char *strlwr (char *str) /* no need for unsigned char */
{
char *p = str;
while (*p) {
*p = tolower(*p);
p++;
}
return str;
}
int main (void) {
FILE *fptr;
char path[PATH_MAX], word[MAX_WORD];
size_t i, len, index = 0, max_words = MAX_WORDS;
/* pointer to allocated block of max_words struct initialized zero */
words_t *words = calloc (max_words, sizeof *words);
if (!words) { /* valdiate every allocation */
perror ("calloc-words");
exit (EXIT_FAILURE);
}
/* Input file path */
printf ("Enter file path: ");
if (scanf ("%s", path) != 1) { /* validate every input */
fputs ("error: invalid file path or cancellation.\n", stderr);
return 1;
}
fptr = fopen (path, "r"); /* open file */
if (fptr == NULL) { /* validate file open */
fputs ( "Unable to open file.\n"
"Please check you have read privileges.\n", stderr);
exit (EXIT_FAILURE);
}
while (fscanf (fptr, "%s", word) == 1) { /* while valid word read */
int iscap = 0, isunique = 1; /* is captial, is unique flags */
if (isupper (*word)) /* is the word uppercase */
iscap = 1;
/* remove all trailing punctuation characters */
len = strlen (word); /* get length */
while (len && ispunct(word[len - 1])) /* only if len > 0 */
word[--len] = 0;
strlwr (word); /* convert word to lowercase */
/* check against 100 most common words (TOP100) */
if (len < TOP_LEN) { /* word less than TOP_LEN? */
char teststr[TOP_LEN * 2]; /* buffer for " word " */
sprintf (teststr, " %s ", word); /* create teststr */
if (strstr (TOP100, teststr)) /* check if in TOP100 */
continue; /* if so, get next word */
}
/* check if word exits in list of all distinct words */
for (i = 0; i < index; i++) {
if (strcmp(words[i].word, word) == 0) {
isunique = 0; /* set unique flag zero */
if (iscap) /* if capital flag set */
words[i].cap = iscap; /* set capital flag in struct */
words[i].count++; /* increment word count */
break; /* bail - done */
}
}
if (isunique) { /* if unique, add to array, increment index */
if (index == max_words) { /* is realloc needed? */
/* always use a temporary pointer with realloc */
void *tmp = realloc (words, 2 * max_words * sizeof *words);
if (!tmp) { /* validate every allocation */
perror ("realloc-words");
break; /* don't exit, original data still valid */
}
words = tmp; /* assign reallocated block to words */
/* (optional) set all new memory to zero */
memset (words + max_words, 0, max_words * sizeof *words);
max_words *= 2; /* update max_words to reflect new limit */
}
memcpy (words[index].word, word, len + 1); /* have len */
if (iscap) /* if cap flag set */
words[index].cap = iscap; /* set capital flag in struct */
words[index++].count++; /* increment count & index */
}
}
fclose (fptr); /* close file */
/*
* Print occurrences of all words in file.
*/
puts ("\nOccurrences of all distinct words with Cap in file:");
for (i = 0; i < index; i++) {
if (words[i].cap) {
strcpy (word, words[i].word);
*word = toupper (*word);
/*
* %-15s prints string in 15 character width.
* - is used to print string left align inside
* 15 character width space.
*/
printf("%-8d %s\n", words[i].count, word);
}
}
free (words);
return 0;
}
Example Use/Output
As was the case last time, your Expected Output: (Example only) is wrong because there is nothing in your code to remove plurals, possessives or plural possessives, so your output with your cars.txt file would be:
$ ./bin/unique_words_exclude_top_100
Enter file path: dat/cars.txt
Occurrences of all distinct words with Cap in file:
2 Motor
8 Cars
1 German
1 Karl
2 Benz
1 Patent-motorwagen
1 Model
1 T
1 American
1 Ford
1 Company
1 Western
1 Europe
1 Electric
2 Road
1 People's
1 China
1 India
Look things over and let me know if you have further questions.
This obviously doesn't work, because it isn't skipping the word if it is a common word like in the misleading comment, but skip the current iteration and continue checking with the next word in the common words list
// skip the word if it is a common word
for (int i = 0; i < NUMBER_OF_STRING; i++) {
if (strcmp(word, commonWords[i])==0) {
continue;
}
}
continue will only affect the innermost loop. Besides, after the loop nothing is changed
To fix that you need to break the outer loop
nextword:
while (fscanf (fptr, "%s", word) == 1) // read the word
for (int i = 0; i < NUMBER_OF_STRING; i++) {
if (strcmp(word, commonWords[i])==0) {
goto nextword; // skip current word
}
}
/// ...
}
Or if you don't want to use goto then another variable must be used
int isCommonWord = 0;
while (fscanf (fptr, "%s", word) == 1) // read the word
for (int i = 0; i < NUMBER_OF_STRING; i++) {
if (strcmp(word, commonWords[i])==0) {
isCommonWord = 1;
break; // exit the for loop
}
}
if (isCommonWord)
continue; // get the next word
/// ...
}
Anyway your implementation is quite inefficient. This is basically a dictionary that maps from a string (the word) to integer (which is the word count). The dictionary can be sorted (like std::map in C++) or hash-based (std::unordered_map in C++). Since you don't sort the array you always have to traverse through the whole list. If the array is sorted then using binary search will cut down the lookup significantly. To check a list of 128 elements you need only at most 7 comparisons instead of 128 like in the case of unsorted list
But before looking for the word in the dictionary you need to check if the word is common or not first. That's done by checking if the word exists in the common word set or not. Again the set can be implemented unsorted (slow), sorted (better, std::set in C++) or hash-based (fastest but needs more memory, std::unordered_set in C++). The difference between the set and the dictionary is that each dictionary entry contains a pair of (key, value), whereas the value is also the key in a set. The for loop checking strcmp(word, commonWords[i])==0 above is a simple set traversing. In any case, once you've found the word in the set, skip the current while loop and not the for loop like I said above. That'll work